[TRACIER - DO NO MERGE] feat: scenario tests for nightly CI runs by redpanda-f · Pull Request #72 · FilOzone/foc-devnet

redpanda-f · 2026-03-03T09:40:51Z

Summary

NOTE: THIS PR IS IN DRAFT SINCE IT IS ALMOST ENTIRELY DECOMPOSED INTO SMALLER PRs

Introduces:

a Python-based scenario testing framework,
a full CI overhaul with nightly,
stability/frontier matrix runs (latesttag testing and latestcommit testing) and automated GitHub issue reporting,
a new latesttag location type for foc-devnet dependentes,
linting infrastructure,
and several bug fixes.

Reviewers may want to look at following Pull Requests for smaller, discrete additions, one at a time:

Base PRs:

Linear, interdependent PRs, to finish up:

Testing infra

Python based testing infrastructure with no dependencies apart from standard python3 libraries.
run.py is a "core", shared testing infrastructure for error reportings and utility functions.
4 tests introduced:
- Added scenarios/test_containers.py — verifies all expected Docker containers are running.
- Added scenarios/test_basic_balances.py — checks Lotus wallet and FEVM account balances.
- Added scenarios/test_storage_e2e.py — end-to-end storage scenario against the live devnet.
- Added scenarios/test_caching_subsystem.py — validates the caching subsystem via CQL in curio.

CI Fixes

Continous Integration needed overhaul for allowing for nightly runs etc.
ci.yml is now:
- ci_run.yml: The core "meat" of the tasks execution
- ci_pull_request.yml is triggered for all PRs raised towards main
- ci_nightly.yml is triggered as cron (
  
  foc-devnet/.github/workflows/ci_nightly.yml
  
  Lines 16 to 17 in fe48197
  
  # Nightly at 03:00 UTC
  
  - cron: '0 3 * * *'
  
  )
  - Two distinct tests are run
  - frontier: Tests for whatever commits are latest on dependent curio, filecoin-services, lotus is locked at latest tag.
  - stability: Tests for whatever latest tags are marked on dependent curio, filecoin-services, lotus is locked at latest tag.

Nightly tags in use:

Issues will be tagged and appended when they have following labels:
scenarios-run-frontier
scenarios-run-stability

Ref:

foc-devnet/.github/workflows/ci_nightly.yml

Lines 36 to 43 in fe48197

    
           - name: stability 
        
             init_flags: "--curio latesttag:pdpv0 --filecoin-services latesttag:main" 
        
             issue_label: scenarios-run-stability 
        
             issue_title: "FOC Devnet scenarios run report (stability)" 
        
           - name: frontier 
        
             init_flags: "--curio gitbranch:pdpv0 --filecoin-services gitbranch:main" 
        
             issue_label: scenarios-run-frontier 
        
             issue_title: "FOC Devnet scenarios run report (frontier)"

`--noterminal` Flag for `version` Command

Added --noterminal boolean flag to foc-devnet version.
When set, suppresses tracing prefixes so output can be captured cleanly by scripts and
scenario reporters.

Bug Fixes & Dependency Updates

Deploy script rename: updated all references from deploy-all-warm-storage.sh to
warm-storage-deploy-all.sh to match the upstream rename in filecoin-services.
- This raises a larger question of what to do when "frontier" has moved to a different setup (codewise in foc-devnet) w.r.t "stability". cc @rvagg
YugabyteDB: removed --advertise_address=0.0.0.0 flag that caused address-binding
conflicts.
YCQL port in devnet-info: added ycql_port field to YugabyteInfo and populated it
from the step context, exposing the CQL port (9042) to consumers of devnet-info.json.
Default dependency versions:
- Lotus: v1.34.4-rc1 → v1.35.0
- Curio: 4d53c80... → e109aec7...
- filecoin-services: 2b247916... → c79e2c6b...

Linting Infrastructure

Added scripts/lint.sh — unified linter running cargo fmt, cargo clippy, and black
in either check mode (FIX=0) or auto-fix mode (FIX=1).
Added scripts/install_precommit_hooks.sh — installs a git pre-commit hook that calls
lint.sh before each commit; supports git worktrees.
Updated .gitignore to exclude __pycache__/ directories and .githooks/.

Pre-merge checklist:

Ensure no github issue has labels scenario-run-* as label
Ensure removal of pull_request trigger here:

foc-devnet/.github/workflows/ci_nightly.yml

Lines 13 to 14 in c477672

pull_request:

branches: ['main']

Post-merge checklist:

Ensure one nightly run was correctly run. Stability may fail for a while till filecoin-services has a tagged commit
Ensure we can run a manual workflow with inputs

rvagg · 2026-03-09T05:21:29Z

scenarios/test_caching_subsystem.py

+
+    if not py or py == "":
+        info(f"Installing custom python version {PYTHON_VERSION}")
+        info(sh(f"npm i --global --silent @bjia56/portable-python-{PYTHON_VERSION}"))


hah what? installing python via npm? pyception. Let' not do this; setup the container's environment in the action, don't install system tools in the test itself

This is very weird indeed. But ycql needs very specific python interpreter and does not like python version that was on my machine (too high of a semver according to it). If we setup the container environment to match python version required here, we are constraining the entire run due to this, and eventually this knowledge may be lost down the line.

Issue here is that we want "scenarios isolation", and portably python seems to me the right way. Agreed, python via npm is not the best. Do you have better alternatives?

added new script, and removed setup in scenarios.

scenarios/test_caching_subsystem.py

scenarios/run.py

scenarios/test_storage_e2e.py

rvagg · 2026-03-09T05:24:40Z

scenarios/run.py

+    _fail += 1
+    sys.exit(1)


eh, I don't think _fail incrementing here is very helpful, do we want to print it maybe?

redpanda-f · 2026-03-09T05:41:19Z

cc @rvagg this action run is a classic case of when "frontier" foc-devnet does not really support "stability" anymore. Till now, I have no good answers for here, apart from doing quite complicated foc-devnet versioning. Would like your opinion

referece actions run: https://github.com/FilOzone/foc-devnet/actions/runs/22837909291/job/66237925146?pr=72

Co-authored-by: Rod Vagg <rod@vagg.org>

redpanda-f · 2026-03-09T06:13:51Z

A couple of things when looking at #75

Can we include the synapse commit that was used as well?

That is scenario specific property, not a property for overall foc-devnet. Logs for the specific scenario has that introduced. ref:

foc-devnet/scenarios/test_storage_e2e.py

Lines 55 to 56 in 8692af2

    
           sdk_commit = sh(f"git -C {sdk_dir} rev-parse HEAD") 
        
           info(f"synapse-sdk commit: {sdk_commit}")

(nit) Status has a period after it "Status." We can remove the period

Removed

For versions info, at the minimum can we remove the prefix that is in the log messages (e.g., "�[32m INFO�[0m �[2msrc/main_app/version.rs�[0m�[2m:�[0m�[2m19:�[0m "). (Even better is just to get it to a simpler bullet list or table, but that's not essential.)

yup, --noterminal flag has been added for this, and should work from next runs. (

foc-devnet/src/cli.rs

Lines 74 to 78 in f610d13

    
               Version { 
        
                   /// Print plain output without tracing prefixes 
        
                   #[arg(long)] 
        
                   noterminal: bool, 
        
               },

)

CI run link: can we link directly do the corresponding "job" rather than just to the "run". For example, link to the stability job.

Introduced, should be available from next runs. ref:

foc-devnet/scenarios/run.py

Lines 309 to 321 in 8692af2

    
           # Print CI run URL in stdout if available 
        
           if os.environ.get("GITHUB_RUN_ID") and os.environ.get("GITHUB_REPOSITORY"): 
        
               github_server = os.environ.get("GITHUB_SERVER_URL", "https://github.com") 
        
               ci_url = f"{github_server}/{os.environ.get('GITHUB_REPOSITORY')}/actions/runs/{os.environ.get('GITHUB_RUN_ID')}" 
        
               ci_url_type = "run" 
        
               # If CI run is available, we may also have the `GITHUB_CI_JOB_ID` set, see `ci_run.yml`` 
        
               # This env var allows us to link to specific jobs instead of entire run 
        
               if os.environ.get("GITHUB_CI_JOB_ID"): 
        
                   ci_url = f"{github_server}/{os.environ.get('GITHUB_REPOSITORY')}/actions/runs/{os.environ.get('GITHUB_RUN_ID')}/job/{os.environ.get("GITHUB_CI_JOB_ID")}" 
        
                   ci_url_type = "job" 
        
               print(f"CI {ci_url_type}: {ci_url}") 
        
           sys.exit(0 if scenario_fail == 0 else 1)

For the test summaries, can we link to the corresponding step as well? For example,I would expect to see a URL with an anchor like #step:39:26 (e.g., https://github.com/FilOzone/foc-devnet/actions/runs/22752589844/job/65990079825#step:39:27)

This is trickier from what I could gather. This needs us to have exact introspective knowledge of where we are in terms of loglines wrt github logging. I think Job link should be enough to navigate.

redpanda-f · 2026-03-09T06:21:17Z

A couple of things:

I assume a better PR description will be provided

yes, added

I was also wondering about how to break this down into smaller units so parts can start getting merged in smaller chunk. I could imagine a PR for the "latesttag" functionality. Maybe another for the githooks and linting, and then the main PR that adds the scenarios?

Many sub items have been raised into separate PRs

I assume we need to add some README_ADVANCED docs about "latesttag"?

Yup, see 0b5816f

BigLep · 2026-03-10T21:36:25Z

@redpanda-f : a couple things

I assume at this point this PR should never be merged. It's serving as an overview of all the functionality that has been added/changed. (This is content that could be in the issue CI/Nightly End-to-End Validation of FOC as a whole #8, but given how this progressed, the content is in here - fine - no problem). I have retitled it to make clear this shouldn't get merged.
For the Curio and filecoin-services dependencies, I think it would be nice to be on tags since we do tagged releases there, but I can get that that we haven't created tags in a bit... chore: updated defaults for config.rs #78 (review)

redpanda-f added 10 commits March 2, 2026 14:49

introduce: initial scenarios

820b67f

feat: githooks, ADVANCED_README

f0fdab5

rename: foc-start-test to foc-devnet-test

803aeb9

add: fixes

930d7eb

add: support for latestCommit and latestTag

d077ef5

fix: scenarios

e6bedf0

fix: test_basic_balances

f7452f6

fix: ensure_foundry was in wrong location

ba87369

feat: latestCommit and latestTag takes a branch optionally

c28437c

init: pythonic tests

a247b2a

FilOzzy added this to FOC Mar 3, 2026

github-project-automation bot moved this to 📌 Triage in FOC Mar 3, 2026

redpanda-f mentioned this pull request Mar 3, 2026

(WIP) feat: scenarios based testing #71

Closed

redpanda-f changed the base branch from feat/redpanda/feat-scenarios to main March 3, 2026 09:43

redpanda-f changed the base branch from main to feat/redpanda/feat-scenarios March 3, 2026 09:44

redpanda-f self-assigned this Mar 3, 2026

redpanda-f added 13 commits March 3, 2026 09:51

add: interbranch PR triggers CI

ce99d99

fix: use '**' glob so CI triggers on slash-separated feature branches

f166fa1

add: interbranch PR triggers CI

f40b6bf

add: interbranch PR triggers CI

d035fa9

remove: shell

d489d84

fix: python tests

676dc42

fix: move into scenarios/ directory

85a2a60

add: README_ADVANCED on how to add pythonic tests

8bcaccb

try: multimachine strategy

02746f3

add: fail-fast: false

f56797c

add: 16xlarge+gpu, 8xlarge+gpu

a8ee429

add: 16xlarge+gpu, 8xlarge+gpu

1758b43

fix: scenarios call

b1b2d94

redpanda-f moved this from 📌 Triage to ⌨️ In Progress in FOC Mar 3, 2026

remove: githooks/, added into .gitignore

fe48197